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The Summer Academic Skills Enhancement Program 
(SASEP) was offered in June-August 1993 to provide Job Training 
Partnership Act (JTPA) clients with the reading comprehension and 
language mechanics skills required for emplojnnent in entry-level 
positions. A total of 125 referred clients were enrolled into both of 
two remedial courses of study. Enrol lees were categorized as 
"regular" (RPA) or "low" pretest achievers (LPA) , and specific 
evaluation objectives were developed for each group. At least 45% of 
the students in the RPA subset showed a 1.0 grade equivalent (GE) 
gain on the reading comprehension and language mechanics posttests. 
Thirty-five percent of the LPA students gained at least 1.0 GE in the 
reading comprehension posttest, and 42X achieve a 1.0 GE in the 
language mechanics posttest. Forty-seven percent of the 1993 program 
participants achieved program completion status (they attended 
classes 75X of the program days). Program retention demonstrated a 27. 
drop from 1992. Male/Black retention (81%) was down only IX from 
1992. It was recommended that all SASEP evaluation levels be 
maintained at their present values but that the definition of program 
completion be relaxed. (Appended are SASEP pupil attendance and 
achievement data. Fourteen tables are included.) (MN) 
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Abstract 

Program Description; The Siunmer Academic Skills Enhancement Program (SASEP) 
was funded by the Private Industry Council (PIC) of Franklin County through 
the Job Training Partnership Act ( JTPA) • The purpose of the program was to 
provide JTPA clients with the Reading Comprehension and Language Mechanics 
skills required for employment into entry** level positions. A total of 160 
curriculum hours were used toward this end. This "Fast Track** program was 
conducted by the Department of Community Education (DCE) , Columbus Public 
Schools* 

Program enrollment was defined by the Private Industry Council. 
Anticipated were 100 PlC**ref erred clients. Sixty ** completions** were 
authorized. A total of 125 referred clients were pretested with the 
Comprehensive Test of Basic Skills (Form U# Level H). These clients were 
enrolled into both of two remedial courses-*of**study!! Reading Comprehension 
(Houghton-Mifflin New Directions in Reading curriculum) or Language Mechanics 
(Houghton«-Mif flin *'whole language" grammar and composition series). 

The 1993 program consisted of two segments: testing (MayJune) and 
remediation (June- August ) . Performance objectives were stated for each of the 
two remediation programs; Reading Comprehension, Language Mechanics. 

Evaluation Design ; 

1. Evaluation Objectives for "Regular Pretest Achievers" rpa 

1.1 45% of the clients enrolled in the reading component who attend 75% of 
the program days and who have a CTBS pretest Reading Comprehension 
grade equivalent score between 5.0 and one less than their age-grade 
placement, will show 1.0 grade equivalent gain on the posttest. 

1.2 45% of the clients enrolled in the language component who attend 75% 
of the program days and who have a CTBS pretest Language Mechanics 
grade equivalent score between 5.0 and one less than their age<-grade 
placement, will show 1.0 grade equivalent gain on the posttest. 

1.3 70% of the clients who attend 75% of the program days and who have a 
CTBS pretest grade equivalent score of 5.0 or more in their component 
area and who score below 75% on the pretest employment skills test, 
will score at or above 75% on the employment skills posttest. (N/A in 
1993) 

*Age-Grade Placement is the grade the client would be in without retention 
in grade. 
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2. Evaluation Obiectives for **Low Pretest Achievera** (LPA> 

2*1 40% of the clients enrolled in the reading component who attend 75% of 
the program days and who have a C7BS pretest Reading Comprehension 
grade equivalent score less than 5*0 will show 1«0 grade equivalent 
gain on the posttest, 

2*2 40% of the clients enrolled in the language component who attend 75% 
of the program days and who have a CTBS pretest Language Mechanics 
grade equivalent score less than 5*0, will show 1*0 grade equivalent 
gain on the posttest« 

2*3 40% of the clients who attend 75% of the program days and who have a 
CTBS pretest grade equivalent score of less than 5.0 in their 
component area and who score below 75% on the pretest employment 
skills test, will score at or above 75% on the employment skills 
posttest. (N/A in 1993) 

3* Other Evaluation Obiectives 

3*1 75% of the clients enrolled will attend 75% of the program days* 

3*2 80% of the male/black clients enrolled in the program will be retained 
through the posttesting phase of the project* 

3*3 The clients served by the program will reflect the target population 
such that 25% of the clients will be non-minority* 

Malor Findings; Pupils in the Regular Pretest Achievers (RPA) subset reached 
criterion level on both Evaluation Objectives (1*1, 1*2)* To be in the RPA 
subset, a pupil had to have a CTBS pretest GE of at least 5*0* The employment 
skills test (Objective 1*3) was not administered in 1993* 

Pupils in the Low Pretest Achievers (LPA) subset failed to reach criterion 
level on Evaluation Objective 2*1 and did achieve 2*2* To be in the LPA 
subset a pupil had to have a CTBS pretest GE less than 5*0* Particularly, 35% 
(N«6 of 17) of LPA clients in the reading content-area gained at least 1*0 GE 
at posttesting* In contrast, 42% of these LPA clients did meet the 2*2 
language criterion level (40% was required to **reach'* this criterion)* The 
employment skills test (Objective 2*3) was not administered in 1993* 

Program retention demonstrated a 2% drop from 1992 (67%, down from 69%; 75% 
was required)* Male/Black retention (81%), down 1% from 1992, was an 
encouraging statistic for the second year in a row* Non-minority 
recruitment/enrollment /retention (N«l) showed a loss from 1992 and is a major 
concern* 

A pupil attains program completion status by attending 75% of the possible 
program day and by showing a gain of one or more grade equivalents (GE) on at 
least one of the three CTBS subtests* In this context, ceiling effect** 
continues to confound assessment of achievement gain and to depress the number 
of clients who achieve program-completion status* As was articulated in the 
1991 and 1992 final reports, numerous clients scored at or above 12*0 (grade 
equivalent on at least one of the three CTBS (UH) tests used to evaluate 
prete8t-*to-posttest change* Such a pretest score defeats the programme 
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purpose on both counts (gain measurement and program completions)* Scni 
recommendation #2 below* 

Fifty-nine clients achieved **program completion** status* This group 
represents about 47% of the 125 pupils enrolled in the 1993 Summer Program* 
Data collected during evaluation of program clients suggest that the number of 
completions could be increased significantly by reducing the attendance 
requirement from 75% (30 days) to about 60% (say, 23 days)* Or, base 
completion solely on whether a pupil completes all pre- and posttests* Either 
way, the 1993 Summer Program would have had at the least d-lO more 
^^completions* ** This is significant because what is being proposed is an 
additional 10-15% more completions* Pupils enrolled for less than 30 days, 
even with perfect attendance, cannot — by definition — become program 
completions, although they may well gain one or more 6S*s on one or more of 
the three tests* 

**Knrollment, ** especially calculation of a percent attending 75% of the 
instructional days offered, is influenced by a large number of pupils who take 
the CTBS pretest and then do nothing else* While it may be correct to use the 
number of clients enrolled as the denominator for calculating "retention** 
data, it appears probable that (under prevailing selection criteria) the 
clientele served by this program is unlikely ever to reach an 80% retention 
level. An alternative algorithm is worth investigating; or, a lower criterion 
value might be considered* 

Recommendat ions 

1. consideration should be given to exposing future SASEP clients to 
instruction in both Reading Comprehension and Language Mechanics* 

2* It is recommended that a new definition of ** completion** be derived and that 
this definition be applied in 1994, in each instance where high (>»12*0) 
pretest CTBS GE values so warrant* 

3. It is recommended that program sponsors and managers test the viability of 
using a ** completion** definition in the future which is either (a) not 
dependent on attendance, (b) is relaxed to, say, 60%, a level that would 
have captured the six pupils lost to attendance in 1993 who did gain at 
least 1*0 GE, and (c) allows for pupils enrolled for less than the full 39 
days of instruction* 

4* Criterion levels for all evaluation objectives should remain at their 
present values for the 1994 SASEP* 

5. Continued emphasis on timely enrollment and daily attendance should 
increase the proportion of 1994 SASEP clients who attain 30 or more days of 
attendance* 

6* Retention of male/black clients should be given high priority for 1994* 

7* Program planners for the 1994 SASEP should structure and implement 
effective, productive strategies for attracting, enrolling and retaining 
non-minority clients* 
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Job Training Partnership Act 

PAST TRACK 

FINAL EVALUATION REPORT 
SUMMER ACADEMIC SKILLS ENHANCEMENT P][\06RAM 

1993 

Program Description 



The 1993 FAST TRACK Stammer Program consisted of two distinct phases s 
testing and remediation. The testing phase was designed to identify pre-test 
performance levels; remediation strategies were adopted to maximize the 
potential for improving clients* content mastery in t%io instructional areas t 
Reading Comprehension (six objectives); (b) Language Mechanics (five 
objectives). The proposed Program Design statement summarized these two 
phases as **Testing** and ''Remediation'*. Underlying the remediation phase was a 
singular goal: provide JTPA clients with the academic skills necessary for 
"employment into entry level positions." 

Employment Skills," an integral part of past PIC Summer Programs, was not 
offered in 1993. 

Referral and Selection 

The target group for this program was defined as "JTPA eligible youths aged 
14-21". 

The selection process was initiated by a referral from the Private Industry 
Council (PIC). Referrals were tested and the Columbus Public Schools* (CPS) 
Department of Community Education (DCE) notified PIC regarding performance 
levels. 

An anticipated 100 PIC-identif ied youth were to be pretested. All PIC 
referrals were enrolled to attend the eight-week remediation phase beginning 
in mid-June* The remediation phase was conducted at the North Education 
Center and emphasized prescriptive/individualized instructional strategies and 
materials. 

Recruitment Methods : The Private Industry Council supplied (by way of the PIC- 
10 referral form) the Department of Community Education with the names of 125 
eligible youth who were chosen or self-identified for participation in the 
Summer Academic Skills Enhancement Program (SASEP). In April, the Department 
administered a CTBS battery to those youth. All 125 clients were selected by 
PIC for the "Fast Track" program. The department also assisted PIC officials 
with orientation and enrollment. 

Testing 

Commencing May 3, 1993 and through June 25, 1993, the Department of 
Community Education administered the Comprehensive Tests of Basic Skills 
(CTBS; 1981), Form U, Level H Reading Comprehension, Language Mechanics, and 
Mathematics Computation subtests to 125 PlC^identif ied youth. The principal 
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assessment activity for students enrolled in the Columbus City School District 
took place at students* home schools. For youth living outside the Colxunbus 
City School District or for youth referred after the testing period was 
closed, testing was administered by PIC personnel at a site designated by PIC 
management • The Department of Community Education supplied PIC staff with the 
test instruments and answer sheets* 

The Department of Program Evaluation (DPE) of the Columbus Public Schools 
^cored completed answer forms and produced individual diagnostic reports and 
system summaries. All scores were norm**referenced. The Columbus evaluators 
used the TESTMATE microcomputer software system to scan, score, and report 
norm-referenced data. 

The Department of Community Education, in concert with the Department of 
Program Evaluation selected Form U, Level H of the Comprehensive Tests of 
Basic Skills (third edition) as the most appropriate level of difficulty for 
the identified client group. The CTBS is a norm-referenced achievement test, 
the content categories of which were defined by examining current state and 
district curriculum guides, published texts and instructional programs, and 
criterion-referenced assessment instrtiments. Columbus evaluation 

professionals selected the Reading Comprehension, Language Mechanics, and 
Mathematics Computation subtests for administration to clients. Total time 
for actual testing was 93 minutes; test administration protocols added 
approximately 35 minutes to the testing session. 

Reading . At the lowest levels, the Reading Comprehension test measures visual 
and sound recognition of letters, words, vowels, and consonants. Items 
measuring comprehension skills are related to sentences and stories. Reading 
Comprehension items measure skills in understanding sentence meaning, passage 
details, character analysis, main ideas, generalization, written forms, and 
author techniques. (Houghton-Mifflin's New Directions in Reading > 

Language Mechanics, These items measure the student's ability to identify the 
correct use of capital letters, periods, commas, exclamation points, question 
marks, quotation marks, colons, semicolons in sentences and in extended 
passages. (Houghton-Mifflin's **whole language** grammar and composition series) 

Testing Methodology Used . The tests* designers used a three-parameter item 

Response Theory (IRT) to scale the CTBS and to develop norms. Application of 
IRT methodology provides a number of direct benefits to the user of CTBS Form 
U, including more accurate descriptions of client performance. Consultants 
from the educational community, represented by native American, Asian, 
Hispanic, and Black ethnic and cultural groups, reviewed all items for 
possible racial, ethnic, and gender bias. Consequently, the standardized 
instruments do not contain items that appeared statistically biased in item 
tryouts. In the standardization, the sample reflects ethnic minorities as 
they are represented in the general population. 

Remediation 

One hundred and twenty-five PIC-referred youth were pretested and enrolled 
in an eight-week summer prescriptive and individualized instructional program 
at the North Education Center as part of the Summer Youth Employment Training 
Program operated by the l>rivate Industry Council. Sixty-seven pupils attended 
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the morning session; 58 pupils attended the afternoon session* The 
instructional phase of the remedial program took place from June 21, 1993 
through August 13, 1993. Those clients who successfully completed course work 
were eligible to receive 0.5 unit of academic credit for reading, or 0*5 unit 
of academic credit for Language Mechanics* 

Clients attended four hours of classes in Reading Comprehension and 
Language Kechanics, Monday through Friday (160 instructional hours). All 
training was delivered by instructors certificated by the State of Ohio. 

At the conclusion of the summer instructional phase, staff administered the 
CTBS (UH) to clients retained to that point. The Department of Program 
Evaluation analyzed data using appropriate descriptive statistics to determine 
whether the summer remedial treatment was effective in improving clients* 
basic academic skills. 

The Department of Community Education chose instructional materials based 
on research findings that have correlated student learning with patterns of 
curriculum organization. Specifically, researchers discovered that highly 
structured instructional formats are most effective when working on basic 
skills competencies with lower achieving students. The following curricula 
were designed to achieve maximtim mastery over a short time tnrough rigorous 
instructional organization: 

Reading Comprehension. The curriculum employed was Houghton-Mifflin *s New 
Directions in Reading program, which has been designed as a Reading 
Comprehension achievement series for high school students who have not yet 
mastered Reading Comprehension skills. The three-part instructional plan 
consists of (a) preparation in vocabulary building, (b) enhancement of 
comprehension skills through guided reading, and (c) review and extension 
exercises to verify comprehension and provide skills reinforcement through 
immediate practice. The comprehension domain is the central focus of each 
instructional unit, and the curriculum stresses 10 comprehension skills: 
understanding punctuation, understanding word referents, using context to 
reveal word meanings, and to understand figurative language, noting important 
details, understanding sequence of events, recognizing the main idea of 
paragraphs, making inferences and drawing conclusions or predicting outcomes, 
understanding causes-effect relationships, understanding comparisons, and 
distinguishing between fact and opinion. In addition to quizzes for 
individual lessons, instructors administered both mid-level and end-of-level 
testing. 

Language Mechanics. The language curriculum used Houghton-Mifflin's **whole 
language*' grammar and composition series. This curriculum integrates grammar 
with reading and writing skills. Grammar units begin with the presentation of 
the basic lesson, and from that base progresses to vocabulary building 
activities. These activities are capped by exercises that assist students to 
make the crucial grammar-writing connection. Students then move to ''checkup'* 
activities that assess mastery levels attained. A cumulative review follows, 
which in turn is supplemented by enrichment work or differentiated additional 
practice (easy, average, or challenging). Reading and writing units commence 
with literature selections and are followed by activities that give students 
practice in using the three modalities of literature response: listening, 
speaking, and thinking (inferring/drawing conclusions). Composition skills 

P:\P552\PICFRT93 
11^24*93 2:23 PM 



8 



f 

I 



4 



are taught through the five**8tep writing process: pre**writing, drafting, 
revising, proofreading, and publishing (final drafting). Students master 
spelling skills using Houghton**Mif flin's spelling program, which supports a 
complete testing program in standardized test format* 



Evaluation Design 

Pretesting of program candidates was used to discern skills deficiencies in 
reading, language, and mathematics. Then, clients were enrolled into either 
or both of two programs: Reading Comprehension or Language Mechanics. 
Program clients were then guided through the remediation phase as described 
above* Clients who completed the eight«-week instructional program were then 
posttested to reveal pre/posttest change with respect to Reading 
Comprehension, Language Mechanics, and Mathematics Computation observed 
scores* 

Because the Stimmer 1991 Program Evaluation design resolved the so**called 
** student ID number problem** successfully, this year care again was taken to 
standardize the number**assignment process* The Department of Program 
Evaluation (DPE) pre-printed CTBS answer forms and prepared lists of students 
who were pretested for use by the Summer Program Coordinator. Siunmer 1993 
program personnel used these lists to code student numbers on all forms 
returned to the DPE for analysis* Thus, it again was possible this year to 
conduct analysis as intended, analyses based on attendance, where 30 days (75% 
of 39 enrollments days) of attendance was the cut*-off point. 

Completion 

Two project-completion criteria were evaluated* To be considered a 
-completion, a client must: 

1. attend 75% (N»30) of the 39 enrollment days; and 

2. demonstrate a grade-equivalent gain (pretest to posttest) of at least 
1.0 on any one of the three CTBS subtests: Reading Comprehension, 
Language Mechanics, or Mathematics Computation. 

Sixty program completions were authorized* 

Evaluation Objectives 

Nine evaluation objectives were stated in the DCS/CPS 1993 Sumer Program 
proposal. These objectives were initiated by the DPE, following review of the 
1992 program. It was suggested that these nine objectives accurately 
represent the program's potential for measuring clients* CTBS scores. But, 
expectations for improving CTBS scores should be tempered by an awareness of 
the wide range of clients* pretest scores and the likelihood of improvement 
therein. 

Thus, evaluation objectives were stated for two groups: Regular Pretest 
Achievers (RPA; pretest grade-equivalent scores of 5.0 or more), Low Pretest 
Achievers (LPA; pretest grade-equivalent scores of less than 5.0). Also, 
objectives regarding Reading Comprehension and Language Mechanics were 
structured for accommodation of ** age-grade placement** (the grade a client 
would be in without having been retained). 
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Two planned evaluations (1*3, 2*3) were not conducted because Employment 
Skills assessment was not included in 1993* 

RPA/LPA Rationale > 

An SASEP is a remediation-type, short-term learning experience* Of 
interest to program planners is whether strategies adopted to guide the 
instructional process work equally well with both RPA and LPA groups. 

Prior to 1992, SASEP data analysis did not include provisions for 
assessment of these two entry-level-split groups* In 1991, following 
examination of 1990 SASEP data the decision to analyze test results with 
respect to RPA and LPA membership was proposed for the 1992 SASEP* This 
analysis plan and the results of its implementation %^re documented in the 
1991 final report submitted to PIC* Because that analysis did reveal some 
insights useful for program-planning purposes, the same analysis model was 
used in 1993. 

The **5.0 GE** demarcation was arbitraxry* It was chosen because observation 
of 1990 data seemed to indicate that **at around this point" would separate 
what appeared to be two groups of clients with divergent CTBS pretest scores 
in reading and language; i*e*, frequency distributions of pretest scores were 
bimodal at approximately G£ » 5*0* 

And, program planners wanted to know how well the Reading Comprehension 
program was working with respect to CTBS-measured pre/posttest reading gain; 
how well the Language Mechanics program was working with respect to 
pre/posttest language gain* Therefore, each group (RPA/LPA) was assigned one 
reading (1.1, 2.1) and one language objective (1*2, 2*2)* 

Age-Grade Placement 

Selection into the RPA group was calculated to include pupils who scored 
higher on the CTBS pretest than those entered into the LPA group, but lower by 
at least 1.0 GE than students performing at grade-level* Thus, a client in 
the RPA group was known to be performing lower than would be expected 
routinely. The rationale used for stxnicturing selection in this way was based 
on the not remarkable fact that these clients should be excellent targets for 
realizing the 1.0 GE gain on one of the three CTBS subtests* 

So, a 9th-grade client scoring a 9.0 GE for Reading Comprehension would be 
at "age-grade." Conversely, a 9th-grade client scoring 8.0 GE for Reading 
Comprehension would be 1*0 behind age-grade* For Evaluation Objectives 2*1- 
2.3, analyses were based on clients scoring 1*0 or more lower than their 
respective age-grade placements* 

1. Evaluation objectives for **Reqular Pretest Achievers** (RPAy 

1.1 45% of the clients enrolled in the reading component who attend 75% of 
the prograun days and who have a CTBS pretest Reading Comprehension 
grade equivalent score between 5*0 and one less than their age-grade 
placement*, will show 1.0 grade equivalent gain on the posttest* 

*Age-Grade Placement is the grade the client would be in without retention 
in grade* 
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1*2 45% Of the clients enrolled in the language component who attend 75% 
of the program days and who have a CTBS pretest Language Mechanics 
grade equivalent score between 5,0 and one less than their age-grade 
placement*, will show 1*0 grade equivalent gain on the posttest, 

1*3 70% of the clients who attend 75% of the program days and who have a 
CTBS pretest grade equivalent score of 5,0 or more in their component 
area and who score below 75% on the pretest employment skills test, 
will score at or above 75% on the employment skills posttest, 

2. Evaluation Objectives for Low Pretest Achievers (LPA> 

2*1 40% of the clients enrolled in the reading component who attend 75% of 
the program days and who have a CTBS pretest Reading Comprehension 
grade equivalent score less than 5,0, v/ill show 1.0 grade equivalent 
gain on the posttest. 

2*2 40% of the clients enrolled in the language component who attend 75% 
of the program days and who have a CTBS pretest Language Mechanics 
grade equivalent score less than 5.0, will show 1,0 grade equivalent 
gain on the posttest, 

2*3 40% of the clients who attend 75% of the program days and who have a 
CTBS pretest grade equivalent score of less than 5.0 in their 
component area and who score below 75% on the pretest employment 
skillfi test, will score at or above 75% on the employment skills 
posttest • 

3. Other Evaluation Objectives 

3*1 75% of the clients enrolled will attend 75% of the program days* 

3*2 80% of the male/black clients enrolled in the program will be retained 
through the posttest ing phase of the project (excluding clients 
expelled) • 

3*3 The clients served by the program will reflect the target population 
such that 25% of the clients will be non-minority. 

Methodology 

one hundred twenty-five prospective clients were pretested with the CTBS. 
Ninety-seven CTBS posttest forms were scored. Demographic and attendance data 
were recorded on revised Pupil Census Forms (PCP) for all youth pretested with 
the CTBS. File folder reports for all 125 youth pretested were computer-* 
generated by the Department of Program Evaluation. These reports were 
customized to include demographic/attendance data and the results of each test 
taken by the pupil, and to record program-completion status. See the Appendix 
for examples of the reports provided by DPS. 

Achievement data were scanned and scored using TESTMATE computer software. 
Demographic and attendance data were encoded by this consultant. Preliminary 
reports to program administration were provided each time new data were added 
to the datasets* These analyses were conducted using an IBM 9375 running CMS. 
This hardware and SAS 6.07 were used to analyze CTBS data in terms of grade- 
equivalent change. 
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At CTBS pretesting (N«125), 58 (46%) were female and 67 (54%) were male. 
Regarding ethnicity, 2 (2%) were non«*mlnorlty and 116 (93%) black. No Asians 
or American Indians were among the 125 clients In 1993 (see Appendix). 
Ethnicity was not provided for seven clients, none of whom were members of the 
evaluation sample. 

At CTBS posttestlng, 84 clients had at least 30 days of registered 
attendance, and were split 34 female (40%), 50 male (60%). Proportionately, 
less females met the attendance criterion than did males. 

The Evaluation Sample 

To be Included In the evaluation sample, a client must have attended at 
least 30 days of Summer Program Instruction. Since **completlon** requires 75% 
attendance (30 of 39 enrollment days), given that achievement gain Is**** 
clearly->->a function of being exposed to elements of a formal Instructional 
process. It was determined that the evaluation 8ample«*->approprlately->->would 
Include only those clients who met this criterion. 

For analysis. It was required that a client meeting the attendance 
criterion also must have pre- and posttest CTBS scores (at least one subtest 
pair). Therefore, only potential **program completions** were Included In the 
analysis. A total of 84 clients met both evaluation sample eligibility 
criteria. 

Major Findings 

The 84 clients (see Appendix) analyzed as the evaluation sample were 
predominantly black, split about 40%, 60% female/male. One of the two non- 
minority clients made the sample. A total of 83 black clients were analyzed. 

Separate analyses of CTBS data were conducted for each of the evaluation 
criteria stated earlier. Because the evaluation sample-«*es8entially — is black 
by ethnic group, further analyses using this variable — ethnicity — were not 
justified. 

Program Completion Analysis 

To be designated a program **completion, ** a client must: (a) attend 75% (30 
days) of the enrollment period; and (b) show a grade-equivalent gain of at 
least 1.0 on one or more subtests (pre/posttest) of the CTBS. The evaluation 
sample included 84 clients with data sufficient to test for program 
completion. (A client missing the CTBS posttest could not become a 
•♦completion, •• by PIC definition). 

This completion-candidate group of 84 clients consisted of 34 (40%) 
females, 50 (60%) males, 83 (99%) blacks; of the black group 33 (40%) were 
female and 50 (60%) were males. All 84 completion candidates were enrolled In 
both instructional areas t Reading Comprehension and Language Mechanics. 

Fifty-nine (70% of the evaluation sample) clients achieved completion 
status, 25 females and 34 males. Fifty-eight of these 59 were black. Twenty** 
five (74%) of black/female completion candidates did reach program completion 
status; 34 (68%) of males/black were completions. 
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A total of 85 **G.B.>«1.0** gains were observed for 84 clients across the 
three content-area types: reading « 30 (36% of 84), language « 41 (49%) and 
math « 14 (17%). 

Eighty-four clients were program-completion candidates (> 30 days of 
attendance and valid posttest minus pretest score)* Fifty-nine (70%) of the 
84 achieved program completion* Among the 84 candidates, 85 difference scores 
>»1.0 were observed; in fact, all 85 difference-score ** completions** were 
observed eunong the 84 clients who achieved program-completion status* This 
same finding was reported in 1992* The correlation between attendance 75% of 
the time and completion in one or more areas remains impressive* 

How does a gain >»1*0 in, for example, reading relate to language and math? 
Consider Table 1, where RCG is Reading Comprehension gain, LMG is Language 
Mechanics gain* RCG, LMG, and MCG » 1 if gain >*1*0* This table also 
includes the variable **COMPL** which is a **1** if program completion was 
achieved. The correlation between reading and language gain >»1*0 is 0*17, 
and the likelihood of pairing program completion with RCG is 0*49; program 
completion and LMG correlate 0.64* None of the RCG/LMG/MCG correlations 
suggest strong relationships* Thus, gain >»1*0 in any one content area 
appears — essentially — unrelated to that in either of the other two areas* 
Data reported here is quite consistent with that reported in 1992. 

The narrative in the remainder of this report references Tables 1-14* The 
following variable names are used in these tables* 



Variable Name 


Comments 


ATTEND 


Number of days of attendance. 


ATT 


Attend 30 days » YBS, NO otherwise* 


PROG 


Instructional curriculum* 


ETHNIC 


Ethnic group* 


GERCA 


Grade equivalent, Reading Comprehension pretest* 


GERCB 


Grade equivalent, Reading Comprehension posttest. 


GELMA 


Grade equivalent. Language Mechanics pretest. 


GELMB 


Grade equivalent. Language Mechanics posttest. 


GEMCA 


Grade equivalent. Math Computation pretest. 


GEMCB 


Grade equivalent. Math Computation posttest* 


AGEGR 


Age-Grade* 


AGEGRPLA 


Age-Grade placement* 


RCG 


Reading Comprehension gain; 




>1*0 GB « YES, NO otherwise* 


LMG 


Language Mechanics gain; 




>1*0 GE « Yes, NO otherwise. 


MCG 


Math Computation gain; 




>1*0 GB « YES, NO otherwise* 


SEX 


Gender; M«male, F«female 


COMPL 


Program Completion l«Yes, NO otherwise 



Table 2 presents pertinent frequencies for the 125 FAST TRACK clients who 
constituted the ** potential** completions group* Table 3 presents both pre- and 
posttest descriptive statistics for the **Base Means**, where base is a term 
used to identify all 125 Fast Track clients as a group* Next, (see Table 4) 
the same variable frequencies displayed for the ** non-completion** subsets* 
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And, Table 5 presents similar frequencies for the ** completion** subset. Table 
6 shows means and other descriptive statistics useful for comparison with the 
base^-means in Table 3« 

Regarding data in Table 6, note the almost nine-day difference in 
attendance. On average. Fast Track clients who completed the 1993 Summer 
Program missed slightly more than three of the 39 instructional days. Hon** 
completion clients missed nearly 13 days, on average. 

Average pretest scores in Reading Comprehension %^re higher in the non- 
completion group than in the completion group. However, pretest-to-posttest 
''gain" (defined in terms of the observed change in grade-equivalent scores 
measured), is — essentially — sero for members of the non-completion group. The 
Reading Comprehension (1.86) and Language Mechanics (2.78) ''gains** for the 
completion group are above 0.50 and well above those observed in 1992. 
Perhaps being in both instructional programs influenced **gain** in both in 
1993. Gain in Math Computation was 0.09 for the completion group. 

Table 7 displays pertinent frequencies for members of the evaluation sample 
(N«84). The sample has only one non-black member (no Asians or American 
Indians). Fifty-eight of the 59 program completions are black; 34 (40%) are 
female and 50 (60%) are male. The common factor, of course, is 30 or more 
days attendance by all members of the evaluation sample. Moreover, 59(70%) of 
the 84 members of the evaluation sample gained at least 1.0 GE on at least one 
of the three tests, an increase of 3% from 1992. 

Evaluation Objectives for "Regular Pretest Achievers** RPA 

1.1 45% of the clients enrolled in the reading component who attend 75% of 
the program days and who have a CTBS pretest Reading Comprehension grade 
equivalent score between 5.0 and one less than their age-grade placement, 
will show 1.0 grade ecjuivalent gain on the posttest. 

A total of 17 clients were in this group. Each of the 17 had an age- 
grade placement exactly one less than age-grade. These are clients who 
were enrolled in both content areas and whose pretest score suggested one 
year of retention. 

Twelve of the 17 possible completions (71%) from this group were 
realized. Ten of the 17 (59%) were Reading Comprehension completions 
(see Table 8), clients who gained at least 1.0 OB from pretest to 
posttest. 

This criterion was reached. 

1.2 45% of the clients enrolled in the language component who attend 75% of 
the program days and who have a CTBS pretest Language Mechanics grade 
equivalent score between 5.0 and one less than their age-grade placement, 
will show 1.0 grade equivalent gain on the posttest. 

This objective is nearly identical to E.O. 1.1, the only exception being 
that the content area Language Mechanics is of interest. Twelve clients 
enrolled in Language Mechanics constitute this group. Of these 12, 10 
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(83%) were program completions. Nine of the twelve (75%) were Language 
Mechanics completions (see Table 9), 

This criterion was reached* 

1*3 70% of the clients who attend 75% of the program days and who have a CTBS 
pretest grade equivalent score of 5.0 or more in their component area and 
who score below 75% on the pretest employment skills test, will score at 
or above 75% on the employment skills posttest* 

The 1993 Fast Track Summer Program did not include Employment Skills 
instruction. E.O. 1*3 was not evaluated* 

Evaluation objectives for "Low Pretest Achievers" (LPA> 

2.1 40% of the clients enrolled in the reading component who attend 75% of 
the program days and who have a CTBS pretest Reading Comprehension grade 
equivalent score less than 5.0 will show 1.0 grade equivalent gain on the 
posttest. 

Seventeen LPA clients enrolled in the Reading Comprehension content also 
had reading pretest-scores of less than 5.0 GE* Of these 17, 10(59%) 
were program completions. But, only 6 of the 10 completions were 
associated with a gain of 1.0 6E or more pretest**to*»posttest in Reading 
Comprehension. Thus, 6 of 17 (35%) met this criterion (see Table 10). 

This criterion was not reached. 

2.2 40% of the clients enrolled in the language component who attend 75% of 
the program days and who have a CTBS pretest Language Mechanics grade 
equivalent score less than 5.0, will show 1.0 grade equivalent gain on 
the posttest. 

Thirty-eight LPA clients from the evaluation sample (45%) were enrolled 
in the Language Mechanics content area, all of whom had a language 
pretest score of less than 5.0. Twenty-five (66%) were program 
completions, 16 (42%) of whom were completions in Language Mechanics (see 
Table 11). 

This criterion was reached. 

2.3 40% of the clients who attend 75% of the program days and who have a CTBS 
pretest grade equivalent score of less than 5.0 in their component area 
and who score below 75% on the pretest employment skills test, will score 
at or above 75% on the employment skills posttest. 

The 1993 Fast Track Summer Program did not include Employment Skills 
instruction. E.O. 2.3 was not evaluated. 



Other Evaluation Objectives 

3.1 75% of the clients enrolled will attend 75% of the program days. 
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Of the 125 clients enrolled in the 1993 Fast Track Summer Program, 84 
(67%) attended at least the minimum of 30 days needed toward program 
completion. This represents a 2% loss from 1992 (see Table 12). 

This criterion was not reached* 

3.2 80% of the male/black clients enrolled in the program will be retained 
through the posttesting phase of the project* 

Sixty-two male/blacks enrolled in one or more program content areas and 
were pretested. Of these 62, only 12 (19%) were not posttested. This is 
an improvement for the second year in a row (see Table 13). 

This criterion was reached* 

3.3 The clients served by the program will reflect the target population such 
that 25% of the clients will be non-minority* 

Only one non**minority (white) client was enrolled and post tested as a 
member of the evaluation sample (see Table 14)* 

This criterion was not reached* 

Summary /Recommendations 

Nine evaluation objectives (E*0*) were proposed for assessment of the 1993 
Summer Academic Skills Enhancement Program ( SASEP ) * Two of these (1*3, 2*3) 
were dependent upon measurement of Employment-Skills items* Because the 1993 
program did not include instruction (or measurement) in the Employment Skills 
area, summer-program assessment was based on the remaining seven £*0*'s* 

These seven E*0**8 were identical to those used for assessment of the 1992 
SASEP; comparison of results from 1992 to 1993 is possible* 

Seventy percent (59 of 84) of the attendance-eligible clients achieved (at 
least) a gain of 1*0 grade equivalent (pretest to posttesx:) on at least one of 
the three measurements (Heading Comprehension, Language Mechanics, Math 
Computation)* This number (70%) represents a 3% gain from 1992* That 59 
SASEP clients became completions speaks well for the program* This represents 
an increase of 26 completions from 1992* Most of this increase, however, is 
probably due to the increased number of clients enrolled into the program and 
who attended 30 or more days of instruction* The evaluation sample in 1992 
numbered 49 (of 71, 69%) and 84 (of 125, 67%) in 1993* Perhaps some of the 3% 
increase in completions is due to the fact that all FAST TRACK clients 
received instruction in both Reading Comprehension and Language Mechanics in 
1993* Empirical def initiveness in this context is beyond the scope of this 
report * 

Consideration should be given to exposing future SASEP clients to 
instruction in both Reading Comprehension and Language Mechanics * 

As was true in 1992 (and before), the issue of ''program completion** arises 
again* Recruitment of clients into the 1993 SASEP included a total of 21 
pupils who scored at least 12*0 GE on one or more of the three CTBS pretest 
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categories. Each instance, of course, makes impossible the opportunity for 
^'completion** to occur in that specific content area. Of these 21, seven 
clients did achieve program completion anyway* Eight of the 14 remaining did 
not achieve attendance completion. Thus, six (attendance completion, no 
program completion) were lost: could not become program completions due to 
high pretest scores (see Appendix). 

These 21 clients totaled 26 pretest GE values of 12.0 or higher. 
Therefore, of the 65 (21*3) possible GE values for these clients only 37 
** completions** really were available. 

These recommendations were presented in the 1992 Final Report and are 
equally important to this report: 

**One thing is certain: '*completions, ** per se> under the current 
mandates, consistently will underestimate positive gains made by 
SASEPs.** 

It is recommended that a new definition of ** completion** be derived 
and that this definition be applied In 1994, in each instance 
where high (>«12.0> pretest CTBS GE values so warrant . 

It is recommended that program sponsors and managers test the 
viability of using a **completion** definition in the future which 
is either (a> not dependent on attendance* (h\ is relaxed to, savt 
60%, a level that would have captured the six pupils lost to 
attendance in 1993 who did gain at least 1.0 GE, and (c) allows 
for pupils enrolled for less than the full 39 days of instruction . 

Both **Regular Pretest Achievers** (RPA) objectives were reached. 
Objective 2.2 was reached (was not in 1992); **Low Pretest Achievers** 
(LPA) were only 5% short on 2.1, an increase of 17% from 1992. Clearly, 
LPA clients fared much better on both objectives in 1993 than in 1992. 

Sixty-seven percent (84) of those enrolled (125) achieved attendance 
completion, 2% less than in 1992 (3.1). Eighty-one percent (50) of the 
62 male/blacks enrolled were posttested (3.2). The one non-minority 
client enrolled, posttested (completion) suggests a lack of recruitment 
effort. 

Criterion levels for all evaluation objectives should remain at 
their present values for the 1994 SASEP . 

Sixty-seven percent (84) of the 125 Fast Track clients in 1993 attended at 
least 30 days of formal instruction. This represents a 5% decrease from 1992. 
Client attendance beginning with day one is important. 

Continued emphasis on timely enrollment and daily attendance 
should increase the proportion of 1994 SASEP clients who attain 30 
or more days of attendance . 

A gain of four percent was made this year in terms of the number of 
male/black enrollments who were retained through posttesting. Clearly, 
progress was made again in 1993. Since about one-half of all enrollments were 
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male/black, it is worthy to reiterate increased emphasis on retention of this 
particular group* 

Retention of male/black clients should be given high priority for 
1994 . 

Enrollment of non-minority clients (N»l) fell to its lowest level 
ever. This program no longer serves a non-minority clientele. The 
recommendation made in 1992 in this regard is repeated: 

Program planners for the 1994 SASEP should structure and implement 
effective, productive strategies for attracting^ enrolling and 
retaining non-minoritv clients . 
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Table 1 



correlation of Program completion and Difference-Score Gain 
(All Variables are Dlchotomous*) 



COMPL 



RCC 



LMG 



MCG 



RCG 0.49 



1.00 



0.17 



0.00 



LMG 



0.64 



0.17 



1.00 



-0.05 



MCG 0.29 



0.00 



-0.05 



1.00 



* N»84 for each variable 



Prepared by 
Office of the Deputy Superintendent 
Department of Program Evaluation 
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PIC 1993 SUMMER PROGRAM 
PUPIL CENSUS FORM 



PIC Stud«nt Nam* 



PIC Room 7/9 



PIC Stud«nt Nufflb«r 



CPS Student Numb«r 





5 



American Indian 



INSTRUCTIONS FOR 



COMPLETING FORM 



1. Enter PIC room numbar . 

2. Enter Columbus PubHc School student number. 

3. Enter grade (grade last year). 

4. verify preprinted values (name, PIC student number, sex, and birth date). 

(Write In changes above preprinted values.) 

5. Circle the appropriate ethnic group. 

6. Enter total days of attendance by this pupil. 

7. Enter total days this pupil was enrolled in the Summer Program this year. 

(Note: days of enrollment must btt greater than or equal to days of 
attendance . ) 

8. In the graphic below, use an "X" (as appropriate) to indicate both 
•^Program- and " Instruct lonal -Content Area" for this pupil: e.g., a 
Dupil in the STEP program receiving Language Mechanics instruction 
Should have an "X" in the row STEP, under the column heading Language 
Mechanics. 

use an "X" to indicate each content area in which this pupil receives 
instruction. An "X" may appear in more than one column but in only 
one row . 





FAST TRACK 
STEP 




If 



FAST TRACK 2 



t 



ERIC 
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PIC Stud«nt Nam* 
PIC Stud«nt Numbsr 
CPS Student Nuinbsr 



PIC 1993 SUMMER PROGRAM 
PUPIL CENSUS FORM 
(F*l«-Fold«r Copy) 

PIC Room 



1 19 



Smx M Grade Q9 Birth Data 052677 

Total Days of Program Attandanca 32 Days. 

Total Days of Program Enroltmant 39 Days. 



ETHNIC GROUP MEMBERSHIP (circlad) 
1 - Non-Minority Q - BlacK 

3 - Spanish Surnama 4 - Asian 
5 - Amarican Indian 



CONTENT AREA 

Raading Languaga Mathematics 
PROGRAM Comprahans 1 on Machan i ca Comou tat 1 on 

Fast Track x x 

Fast Track II 

Stap 



C T B S TESTING RESULTS 
(Grada Equivalent Values) 



Reading Comprehension 
Language Mechanics 
Math Computation 



Pretest Posttest Change 

4.0 4.0 0.0 

4.0 4.0 0.0 

4.6 4.3 -0.3 



« 






« 


« 
« 






« 


« 
« 


Cri ter Ion 


Status 


« 
« 


« 
« 


Attendance (30 Day Min. ) 


YES 


« 
• 


« 
« 


CTBS Grade-Equivalent Gain (GE>«1.0) 


NO 


« 
« 
« 


« 
« 


Program Completion 


NO 


« 
« 


« 




* 



«««««««««««««««««« 



Prepared by 
Office of the Superintendent 
Department of Program Evaluation 
(picprog foldrpcf) 

Run Date: 24AUG93 
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